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Abstract 

Due to various kinds of uncertainties, a robot 
motion may fail and result in some unintended con- 
tact between the object held by the robot and the 
environment, which greatly hampers robotics appli- 
cations on tasks with high-precision requirement, 
such as assembly tasks. Aiming at automatically 
recovering a robotic task from such a failure, this 
paper discusses, in the presence of uncertainties, 
contact detection based on contact motion for re- 
covery. It presents a framework for on-line recog- 
nizing contacts using multiple sensor modalities in 
the presence of sensing uncertainties and means for 
ensuring successful compliant motions in the pres- 
ence of sensing and control uncertainties. 

1 Introduction 

The issue of detecting and recovering errors of robot ac- 
tions due to uncertainties (e.g., mechanical, control, model- 
ing, manufacturing, and sensing uncertainties) is crucial for 
robotics applications on tasks with high-precision require- 
ment, such as assembly tasks. Since errors of a robot action 
almost always lead to some unintended collisions between 
the object moved by the robot and some other objects, on- 
line recognition of those collisions or contacts is extremely 
important to recovery strategies. On the other hand, re- 
covery motions are usually preferred to be contact motions, 
i.e., compliant motions, in order to reduce the effect of un- 
certainties via the physical constraints among objects. 

The contact detection problem not only requires sens- 
ing and sensor-based reasoning but also demands them in 
greater precision with sensing uncertainties being taken into 
account. Fig. 1 shows an example to illustrate this. A 
robot is used to perform the peg-in-hole task as depicted in 
Fig. la. Due to uncertainties, the peg may hit somewhere 
other than the desired goal, as in the two cases shown in 
Fig. lb and c, respectively. If the peg in Fig. lc only leans 
very slightly towards the wall, then the contact may not 
be distinguishable from the one in Fig. lb due to sensing 
uncertainties (e.g., position/orientation sensing uncertain- 
ties). Nevertheless, the recovery strategies for the two cases 


should be different. The recovery motion for the case in 
Fig. lb can simply be a compliant translation, while for the 
case in Fig. lc, the recovery motion should also involve ro- 
tation. Thus, the two cases have to be distinguished. On 
the other hand, not all the details about a contact are im- 
portant to recovery motions. For example, the cases shown 
in Fig. lb, Fig. Id, and Fig. le are different in terms of 
the relative locations of the objects in contact and the pre- 
cise topological relationships among the surface elements of 
those objects. Nevertheless, the recovery motions of those 
contacts may follow the same strategy — a compliant trans- 
lation along the surface of contact towards the hole. 

The research directly targeted to contact detection in 
the presence of uncertainties can be found in the work by 
Desai etc. [3, 4] and by Spreng[6]. Both approaches are of 
hypotheses-and-tests kind, i.e., testing the validity of cer- 
tain contact hypotheses to obtain the correct contact in- 
formation. Desai ’s method, in particular, first assumed a 
set of possible contact formations (between two objects), 
and then used force/moment equilibrium conditions with 
the force/moment sensory data to eliminate certain contact 
formations. However, the key problem of how to obtain the 
contact hypotheses (i.e., initial contact formations) remains 
intact. Spreng’s method used position/orientation sensing 
data to hypothesize about a contact in terms of motion free- 
doms and test motions for verification. The method, how- 
ever, seems to be limited to 2D cases. Moreover, the use of 
test motions may cause new failures and further complicate 
the situation. 

Although recovery motions are apt to be compliant to 
be less sensitive to uncertainties, the effect of uncertainties 
must still be taken into account in order to ensure success- 
ful implementations of the desired compliant motions. For 
example, in order to push the object in Fig. 2a along the 
surface successfully, the applied force must be in proper di- 
rection (w.r.t. the normal N of the surface) and magnitude 
to overcome the friction as well as to keep the object al- 
ways in contact with the surface. This, however, has to be 
achieved in the presence of the orientation sensing uncer- 
tainty in N, the force/moment sensing uncertainty, and the 
modeling/control uncertainty in the force controller. So far 
the problem has not been addressed in the literature. 
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Figure 1: A Peg-in-Hole Example 



Figure 2: Compliant Translations 

This paper will first discuss what kind of contact in- 
formation is enough for planning recovery motions, an is- 
sue that has not been addressed in previous research, and 
define recovery-oriented concepts of contact. Then it will 
present a framework of using different sensors, especially 
position/orientation and vision sensors, to compensate each 
other in order to obtain the contact information of desired 
precision in spite of sensing uncertainties. It will also ex- 
plain how to ensure the success of compliant motions by im- 
posing proper force/moment constraints on the commanded 
force/moment applied to the held object (by the robot) and 
certain design constraints on the nominal and uncertainty 
parameters of the system. 


2 Contacts and Assumptions 

Since the contact detection problem mainly deals with un- 
expected interactions between the object held by a robot 
and the environment w T hich, in most cases, is known ap- 
proximately, we can assume that the environment is fixed in 
the sense that all parts or fixtures in the environment are 
pre- known; only the collisions between the held part and 
other parts can be unexpected. Thus, the problem can take 
advantage of a relatively stable and knowm environment in 
contrast to a robot navigation problem. We can also assume 
that the objects involved in an unexpected collision (i.e., 
the held object and some fixed objects in the environment) 
are in a static state, provided that there are force/moment 
guards to stop a robot motion once a collision occurs. 

Now the concern is what kind of contact information will 
be needed in providing enough aid to the planning of recov- 
ery motions. From the example shown in Fig. 1, one can 
see that the detection of contact surfaces is surely impor- 
tant since they constitute the constraining surfaces for the 
compliant recovery motion. In addition, the basic topologi- 
cal formation of contact also matters since different forma- 
tions may require different courses of recovery motion even 
if the contact surfaces are the same (as shown by the two 
contact cases in Fig. lb and Fig. lc). However, not all the 
details in the formation of a contact are important to recov- 
ery motions (e.g., the cases show r n in Fig. lb, Fig. Id, and 
Fig. le share the same kind of recovery motion). Thus, we 
will introduce the concepts of contact which both facilitate 
detection and meet the need of recovery planning. 

We will use the following topological exterior-elements 
of objects: faces, edges, and vertices in our descriptions. We 
define a face as a closed surface, i.e., a surface and its bound- 
aries, and an edge as a closed edge line, i.e., an edge line and 
its boundary points 1 . Clearly, the exterior of a finite solid 
consists of finite faces with shared boundaries (among two 
or more faces). The boundaries of a face consist of edges, 
and the boundaries of an edge consist of (two) vertices. We 
say two topological elements touch iff the interior region of 
the two elements touch. Thus, we don’t think that an edge 
touches a face if only a boundary point of the edge (i.e., a 
vertex) touches the face, and instead, we say that a vertex 
touches a face. 

We now define a principal contact (PC) between two 
faces as one of the following: face- face (f-f), face-edge or 
edge- face (f-e or e-f), face- vertex or vertex- face (f-v or v- 
f), edge-cross (e-cross), edge-touch (e- touch), edge- vertex or 
vertex-edge (e-v or v-e), vertex- vertex (v-v) (Fig. 3), such 
that if there are more than one pair of topological elements 
(from the two faces respectively) that touch each other, the 
PC is determined by the pair in which the two topological 
elements are not the boundaries of the topological elements 
(of their respective faces) in the other pairs in touch. Now a 
contact formation (CF) can be introduced to define a con- 
tact between two objects, as a set of PC’s involved (e.g., 

{< fhfi >.< e l/i 2 >. - ••})*• 

1 Formal definitions of surfaces, edge lines , and vertex points can be 
found in[10]. 

2 This definition is quite different from that in[4, 3]. 
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Figure 3: Principal Contacts between Two Faces 

It is not difficult to observe that, for polyhedral objects, 
a PC of type f-f, f-e (or e-f), f-v (or v-f), or e-cross, involves 
only one contact plane (CP) — the tangent plane through 
the contact points determined by the principal contact. For 
non-polyhedral objects, a PC of type f-f, f-e (or e-f), f-v (or 
v-f), or e-cross, involves either a contact surface determined 
by the principal contact points or a contact plane tangent 
to the principal contact point(s). 

It is, on the other hand, not generally proper to talk 
about a contact plane or surface for a PC of e- touch, e-v 
(or v-e), or v-v type, since there exists an infinite number 
of such contact planes or surfaces. However, PC’s of types e~ 
touch, e-v (or v-e), and v-v are rather purely mathematical 
concepts and rarely occur in reality due to their extremely 
unstable nature. Thus, we will not consider those types 
in this paper, assuming that they have zero probability of 
occurring. For simplicity, we will also restrict our discussion 
to polyhedral objects; thus surfaces are reduced to planes 
only, edge lines are reduced to straight-lines , and there is a 
contact plane associated with each PC. 

3 Contact Detection 

Let’s consider an unexpected collision result in a contact 
between the object held by the robot obj h and one fixed 
object. In principle, if the location of the held object can 
be sensed, then the contact formation can be deduced or 
derived from that sensed location, the boundary represen- 
tations of both objects in the CAD model- base, and the 
pre- known location of the fixed object. However, in prac- 
tice, the sensed location is often different from the actual 
location due to sensing uncertainty; thus, the contact for- 
mation derived can be wrong, or the derivation yields no 
contact at all. To solve the problem, our proposed strategy 


is to first obtain all possible contact formations based on 
the current locations sensed about the two objects, taking 
into account position/orientation sensing uncertainties, and 
then to use vision sensing to reduce the set of contact for- 
mations and to obtain satisfactory information about the 
contact. Force/moment or other sensing methods can also 
be included in the system. 

3.1 Position/Orientation Sensing 

First, the fixed object can be identified fairly accurately 
based on the sensed location of objh, since the contact was 
due to the motion deviation of obj h from a preplanned path 3 , 
and the deviation is generally small. Suppose the fixed ob- 
ject identified is obj/. Then, the location of objj can be 
obtained from the pre-stored database. Let //, e/, and vj 
denote the face, edge, and vertex items of objj respectively, 
which are described with respect to the coordinate system 
of objj , Given the location of objj in the reference coor- 
dinate system of the workspace, i.e., the world coordinate 
system, those descriptions can then be easily transformed to 
be with respect to the world coordinate system. Similarly, 
let f h , e^, and v h indicate the face, edge, and vertex items of 
obj h , described in the coordinate system of obj h . Based on 
the sensed location of objh. in the world coordinate system, 
those descriptions can be transformed to be with respect to 
the world coordinate system. 

Now we need to examine how the information of the 
given location of objj and the sensed location of obj \ can 
contribute to the detection of the contact formation be- 
tween objj and objh ■ Let e P denote the position sensing 
uncertainty , such that for any point p, iip° - pii < 
where P a and P 5 are the actual and the sensed positions of 
P. Let the angle e 0 denote the orientation sensing uncer- 
tainty, such that for any vector N, Z(N a ,N J ) < e 0 , where 
N a and N J are the actual and the sensed vectors. Obvi- 
ously, the uncertainties c p and e Q in the location of objh 
affect the descriptions of fh, e^, and Vh items in the world 
coordinate system and thus the determination of the spatial 
relationships between those surface elements of obj h and the 
surface elements of objj. Fig. 4 gives an example showing 
the ambiguity in determining a PC due to t p and e Q . It 
is not difficult to observe that while there are many possi- 
ble PC’s, the possible contact planes involved are fewer. In 
other words, contact planes are relatively robust and insen- 
sitive to position/orientation sensing uncertainties. There- 
fore, we use position/orientation sensing to reason about 
contact planes first. The objective is to determine all pos- 
sible contact planes, and for each contact plane, all pos- 
sible PC’s that may contribute to it, based on the given 
location of objj , the sensed location of objh, and the posi- 
tion/orientation sensing uncertainties e p and e Q . 

The detection of possible contact planes can be done by 
checking the relationship between a face of objh and a face 
of objj for all possible pairs of such faces between the two 
objects. Consider a face /jj of objh and a face fj of objj , 

3 Therefore, the sequence of the objects adjacent to the path is 
known. 
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Figure 4: The Uncertainties in PC due to t p and t 0 

which lie on planes p\ and p* f respectively. If the projection 
of fl on p 3 j possibly intersects fj , taking into account and 
c 0) then we can check if the minimum distance d-min between 
the sensed f' h and fj is greater than c p or not. Fig. 5 shows 
examples of different spatial relationships between and 
fj and the corresponding d m i n for each case. If d m ^ n > t p , 
we can conclude that there is no contact between f^ and /y. 
Otherwise, there exists the possibility of a contact between 
fl and fj. The next step is to determine all the possible 
contact planes and PC’s between the two faces. Our strat- 
egy is to construct models of all possible PC’s by virtually 
(not physically) conducting the following operations on f' h 
and fj: 

• TOUCH — translate f' h (or fj) along d mtn a distance 
d min to make f' h (or fj) touch fj (or f' h ), 

• JILT — tilt f{ (or fj) about some axis on pj coincid- 
ing an edge or vertex of f x h (or fj). 




Figure 5: between f' h and fj 

Specifically, if d min < e„, TOUCH will be conducted to 
make f' h contact fj, and a PC can be determined by the 
orientations of /£ and fj. The relationships between //, and 
fj can be distinguished in the following ways (Fig. 5): 

i- n li fh 

2. case 1 does not hold, and some edge e of one face is 
on the plane of another face; 

3. both 1 and 2 do not hold, and all vertices of one face 
are on the same side of the plane of the the other face; 

4. none of 1, 2 and 3 hold. 

Based on which case holds, the PC can be of types f-f, e-f 
or f-e, f-v or v-f, and e-cross respectively. 

Based on the result of TOUCH, which gives a contact 
plane and a PC, TILT can be applied to vary the orienta- 
tions of f' h and fj within the orientation sensing uncertainty 
bound e 0 so that other possible PC’s and contact planes can 
be obtained. Fig. 6 shows some examples. Note that the 
fundamental issue about TILT is how to tilt in order to get 
all possible PC’s. There are generally an infinite number 
of ways of tilting f[ or fj with the variations of orientation 
maintained within the range of t Q . However, since the vari- 
ations can only result in a finite number of PC’s, just such 
number of tiltings will be sufficient. The definition of TILT 
above reflects this observation. For example, if the initial 
PC is < fl, fj >, we can tilt f' k or fj along all its edges and 
each line through one of its vertices on the contact plane 
which is not collinear to the two edges forming the vertex; 
then we will obtain all possible e-f (or f-e) and v-f (or f-v) 
types of PCs. Based on each e-f (or f-e) PC, if the edge 
intersects an edge of the other face, then by tilting about 
the latter edge, a possible e-cross PC can be obtained. 

By considering all possible face pairs between obj ^ and 
objj in the way described, while trying to avoid or eliminate 
redundancy, the possible contact formations between obj ^ 
and objj can be obtained. The final result would contain a 
set of possible contact planes (CP), and for each CP, a set 
of possible PC’s that may result in the CP. 
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Figure 6: Possible PC’s obtained by TILT 



Figure 8: A Picture to Show if fj 



and ft Are in Contact 


3.2 Vision Sensing 

Using vision sensing to identify contacts is attractive in 
that the image information of a contact can convey most 
directly the topological meaning of the contact, i.e., the con- 
tact formation. As introduced previously, due to the posi- 
tion/orientation sensing uncertainties, it is difficult to deter- 
mine whether a basic contact formation between objh and 
objj really exists given the location of objf and the sensed 
location of obj h . For example, in Fig. 7, it is impossible to 
know which contact formation the contact is really in from 
position/orientation sensing only, if the horizontal distance 
between objf and obj ^ is smaller than e p . By vision sensing, 
however, the problem could be solved if certain picture(s) 
could be taken properly and reasoned effectively. For exam- 
ple, if a picture can be taken from the direction opposite to 
the normal of /j, then whether fj and ft are in contact can 
be determined by checking whether fj touches ft in the im- 
age (Fig. 8). Obviously, the following issues are important 
in using vision: 

• how to view the contact, i.e., how to place the camera 
and take a picture; 

• how to separate the features of interest from the rest 
of the things in an image; 

• how many different views should be taken in order to 
obtain sufficient information about a contact. 



Figure 7: The Ambiguity in Contact Formation 


The information obtained from position/orientation sens- 
ing is essential for dealing with the above issues. Recall 
that from position/orientation sensing, all possible contact 
planes can be obtained and for each contact plane, all possi- 
ble PC’s that may contribute to the contact plane can also 
be obtained. This information not only defines the goal of 
vision sensing: to eliminate non-existing contact planes and 
the non-existing PC’s, but also provides clues on how to do 
it. 

To view a contact, pictures can be taken based on each 
contact plane and the associated PC’s which are the result 
of processing position/orientation sensing data. We can as- 
sume that a camera is held by another robot hand so that 
it can be placed in different locations easily. Then the topo- 
logical surface elements that appear in a picture and con- 
tribute to a PC (and the contact plane) can be extracted 
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from the picture by combining image segmentation/labeling 
techniques with the 3D modeling infoirniation and the sensed 
position /ovient at ion of those elements. Since the processed 
image will only contain simple surface elements in the forms 
of 2D edges and vertices, it will be easy to reason about 
their relationship. With several images taken from different 
views, one can expect to obtain sufficient information about 
a PC. In the following paragraphs, we will describe a strat- 
egy to detect a PC using vision. By this strategy, what we 
want to know from an image will be a simple fact such as 
whether the concerned surface elements of two objects are 
in a line contact, in a point contact, or in no contact, and 
that information can be quite reliable in spite of noise (or 
uncertainties) in the image. We assume that proper thresh- 
olds can be easily found based on the size of the objects 
and the (bounded) noise to determine whether a contact is 
a line or a point . 

To check if a given contact plane really exists, one can 
place the camera in a way such that the image plane is 
perpendicular to the contact plane (see Fig. 8, where the 
contact plane is determined by fj and /jj). By processing 
the image so that it contains only the edges of fj and 
whether the contact plane really exists can be determined 
easily. 

To eliminate the wrong types of PC s from a given set 
of possible PC’s of a contact plane, one can take pictures 
for each possible PC and check if the result is as predicted. 
If not, the PC can be eliminated. Specifically, to confirm a 
f-f PC < /£, fj >, four pictures can be sufficient (Fig. 9): 

• picl — taken along the contact plane in a direction 
orthogonal to an edge of /£; 

• pic2 — taken along the contact plane in the direction 
orthogonal to the direction of picture 1; 

• pic3 — taken along the contact plane in a direction 
orthogonal to an edge of fj ; 

• picA — taken along the contact plane in the direction 
orthogonal to the direction of picture 3. 

If all the pictures (which should be segmented and labeled as 
described previously) show that the contact region between 

the face elements of f' h and fj forms a line, then < flifj > 
is confirmed. In some cases two pictures can be sufficient. 
For example, if there is no f-e (or e-f) type of PC’s for the 
given contact plane, then we only need picl and pic2 to 
confirm the PC < fk,fj >• To confirm a f-e (or e-f) PC 
< f' h ,e > (or < e,/j >), the following two pictures can be 
sufficient (Fig. 10): 

• picl — taken along the edge e on the contact plane; 

• pic2 — taken along the direction perpendicular to e 
along the contact plane. 

If pic2 shows a line contact region between the relevant face 
elements of the two objects, while picl shows an approxi- 
mate point contact (or a much shoHer line contact as the 



Figure 9: Four Pictures to Confirm a f-f Type PC 



Figure 10: Two Pictures to Confirm a f-e or e-f Type PC 



, ' contact plane 


Figure 11: Two Pictures to Confirm a f-v or v-f Type PC 


effect of orientation sensing uncertainty in e), then < /£, e > 
(or < e,/j >) is confirmed. Similarly, two pictures are suf- 
ficient to confirm a f-v (or v-f) PC < f' h ,v > (or < u, fj >). 
If two pictures are taken along the contact plane in orthog- 
onal directions towards the vertex v (Fig. 11), and the con- 
tact regions shown on both pictures are points, then the PC 
< /j,t? > (or < v,fj >) is confirmed. As for an e-cross PC, 
since it will not share a contact plane with other type of 
PC’s, if the contact plane exists, the PC is confirmed. 
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4 Integration of Other Sensors 

We have shown that by using vision to eliminate the non- 
existing contact planes and the non-existing PC’s, the ex- 
act contact forrflation can be determined from the set of 
possible contact formations initially obtained from posi- 
tion/orientation sensing. However, the major limitation of 
vision sensing lies in the possible occlusion of certain PC’s, 
especially when the objects are non-convex. Thus, other 
sensing means, such as force/moment and tactile sensing[3, 
4][1, 2] may also be needed, which can be readily added in 
this stage. 

5 Compliant Motions for Error Re- 
covery 

A compliant recovery motion of the held object from an un- 
expected contact can be automatically plannedfll] with the 
detection of the contact formation and the contact planes 
involved, as well as other information, such as the desired 
path of the held object and its current (sensed) location. 

For polyhedral objects (as assumed in Section 2), there 
are the following basic types of compliant motions: 

• translations constrained by one plane or two planes 

(Fig. 2), 

• frictional rotations (Fig. 12a), 

• non-frictional point-constrained and line-constrained 
rotations (Fig. 12b), 

• combined frictional/non-frictional rotation (Fig. 13a), 
and 

• combined translation/frictional-rotations (Fig. 13b and 
c). 




Figure 12: Compliant Rotations 





It is necessary to determine then, in the presence of un- 
certainties (as introduced in Section 1), proper forces and 
moments to be applied to the held object by the robot, so 
that, based on the contact information (contact formation 
and contact planes), each type of the above motions can be 
implemented successfully in spite of uncertainties. 

For pure compliant translations and rotations (as listed 
above), detailed analysis can be found in[9, 8] in which the 
proper forces and moments are determined in terms of force 
and moment constraints involving uncertainty bounds and 
under certain design constraints of system parameters. Note 
that the orientation sensing uncertainty is modeled as t 0 in 
Section 3.1. of this paper. The imperfections associated 
with force/moment sensing, modeling, and control are mod- 
eled simply as force/moment control uncertainties , ejj and 
e mm , which are defined as the maximum possible difference 
in magnitude between a desired or commanded force and the 
actual force applied and the maximum possible difference in 
magnitude between a desired or commanded moment and 
the actual moment applied respectively. 


Figure 13: Combined Compliant Motions 



Figure 14: Force/Moment for a Combined Motion of 

Translation/Frictional-Rotation 
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In the same fashion, constraints can be derived for com- 
bined compliant motions, for example, a combined motion 
of translation and frictional rotation (as depicted in Fig. 13b). 
Such a motion can be implemented by applying a force F 
and a moment M c on the center of the object as shown in 
Fig. 14a. The combination of F and M c will result in com- 
pliant translations of the contact points of the held object 
against frictions 4 , where f\ and f 2 are the equivalent forces 
generating the compliant translations. However, due to c 0 
in the sensed normal of plane 1 and ejj, the actual ap- 
plied force F a may not be parallel to the actual normal nj. 
Similarly, due to c 0 and c mm , the actual applied moment 
may not be exactly parallel to the actual intersection 
line of plane 1 and plane 2. It is thus necessary to dis- 
tinguish the force/moment components that will generate 
the desired translation/frictional-rotation from the compo- 
nents that may cause undesirable motions of the held ob- 
ject. Upon the force/moment components for generating 
the translation/ frictional-rotation, constraints can be de- 
rived involving e 0 , Cyy, e mm , and the friction coefficient p, 
which when satisfied, guarantee that no sticking will occur 
and that the motion will always be compliant (i.e., contacts 
will always be maintained). On the other hand, upon the 
force/moment components that may cause undesired mo- 
tions, constraints (also involving e 0 , €//, e mm , an d p) can be 
derived so that when they are satisfied, the effect of friction 
will prevent the undesirable motions frem occurring. By 
synthesizing the two sets of constraints obtained, proper 
constraints on the magnitudes of the commanded F and M 
can be obtained, as well as possible design constraints on e a , 
p, and other object- related parameters (such as 
those characterizing the shape and size of the held object). 
Upon the satisfaction of the design constraints, and by chos- 
ing proper F and M based on the force/moment constraints, 
the desired translation/frictional-rotation can be achieved 
in spite of uncertainties. As for the force/moment control 
to implement a desired force/moment (which is determined 
by our force/moment constraints), many approaches can be 
found in the literature, as have been surveyed and compared 
by Hollerbach[5] and Whitney [7]. 

6 Conclusions 

This paper studied the effect of uncertainties in recogniz- 
ing failures of robot motion in the forms of contacts and in 
implementing contact-based recovery motions. It proposed 
recovery-oriented contact detection using multiple sensing 
modalities and outlined means to ensure successful contact 
motions in spite of uncertainties. The research, however, is 
in its early stage. Further development and testing of the 
idea are necessary in the future. Subjects of special im- 
portance includes studying how accurate vision information 
will be and uncertainties in vision, investigating further in- 
tegrations of different sensors in the system, and testing the 
existing and new results. 

4 Let p be the friction coefficient of the materials, and assume the 
Coulomb friction cone model. 
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